Software testing is an ongoing task in computer software program development and maintenance which requires a large portion of development time, computer and human resources, and effort. Software development may include the development of an entirely new application or program, or the addition of a new feature to an existing application. Software maintenance activities generally include the correction of reported problems.
Testing is performed with the goal of verifying the correct functioning of new software and modifications to existing software. Generally, software testing accompanies even minor code modifications or enhancements to ensure correctness. Verifying the correctness of software may involve numerous tasks ranging from ensuring correct coding syntax through successful compilation, to checking the execution results by examining the output of a software program.
In order to test the execution of software, a machine-executable program comprising binary instructions and data in machine-readable form must be produced and executed. The software may be written in some high-level or low-level programming language. This software may be processed by a compiler, language processor, or translator to produce an object file containing binary or machine-readable code. Usually, there are multiple software modules or source files which comprise the software to be tested. Each of these software modules may be separately compiled, and a separate object file may be produced for each of these source files. These object files may be combined into a single machine-executable program using an operating system tool such as a linker which links together the multiple object files containing binary code and data to produce a single combined machine-executable program containing binary code and data. This machine-executable program may be run or executed on a computer system, and the results from the execution may be examined as a means to ensure correct functioning of the software.
Software programs vary in complexity and size. Both the small and simple programs as well as the large and more complex programs have a need for efficient software testing. Generally, as complexity and size increase, the amount of testing and the need for efficient testing increases as well.
Complex software programs may require development coordination effort due to scheduling dependencies of dependent software units and conflicting requirements for computer resources. For example, a software program may have certain functional components being developed in which each component requires the completion of multiple software units. To test a component requires the simultaneous completion of multiple software units. The total time required to test a component may be reduced by shortening the testing time required for the software units which comprise the component. In such a case, there tends to be a ripple effect of dependencies. By reducing the amount of testing time required for a critical software unit upon which other software units depend, the overall development time for a complex program may be reduced.
The development of a large software program is usually done by multiple developers. Generally, the testing methodology employed in such cases begins with individual developers testing their own code in unit testing. For example, each software component under development may comprise multiple software units each being completed by a different developer. Each developer successfully completes unit testing to verify the correctness of her own unit code independent of other newly developed code units. Unit testing typically follows a frequent iterative test-retest process. This test-retest process includes initially testing code, making code modifications or adding new test data, and then retesting the code.
A problem in performing unit testing arises due to external dependencies upon other untested code. Such external dependencies may prohibit a developer from successfully completing compiling and linking of the software unit which is being tested. For example, a code unit "A" may perform calls to routines in other code units "B" and "C". When "A" is unit tested, it is necessary to identify and resolve its dependencies on units "B" and "C" to successfully compile and link code unit "A" and obtain a machine-executable program which may be executed as part of unit testing "A". Thus, such external dependencies need to be resolved to successfully compile and link a code unit to be tested in order to obtain a machine-executable program. The machine-executable program may be executed to obtain test results which are examined to verify the correct execution of the software unit.
Following successful unit testing, integration testing may be performed. Integration testing comprises incrementally combining and testing multiple software units. For example, multiple software units comprising the software developed for a component may be combined in integration testing. Problems arise during integration testing which are similar to those which arise during unit testing. As software units are incrementally combined, the external dependencies may change. A developer may determine external dependencies through manual code inspection, compiling, or linking errors, and may resolve dependencies by a manual coding solution. This is time consuming and inefficient. Similar to unit testing, an efficient way of identifying and resolving these external dependencies is needed so that integration testing can be completed expeditiously and without undue difficulty.
Additionally, it is desirable to have consistency and uniformity in unit testing methods, test input data, and test results for ease and efficiency in the integration of software units and corresponding test data and results that may be combined as part of integration testing.
Similar software testing problems arise in the maintenance of existing software applications.
There is also a need in both new software development and software maintenance for an efficient way of verifying the correctness of a solution. Once one problem has been corrected, it is necessary to insure that no regressions occur as a result of further code modifications or additions. Regressions occur when new or modified code, for example, corrects one problem but causes other problems which did not exist prior to the addition of the new or modified code.
In order to minimize regressions, once the output from executing a set of software modules has been verified for given input data, it is desirable to use both the verified input data and corresponding output in subsequent testing to verify correctness.
A quantitative problem with testing all software is choosing appropriate metrics and an efficient means of obtaining the metrics to determine when testing is sufficient, and what additional tests may be needed for sufficient testing. Due to the variety of existing testing approaches, testing criteria, and methods, it may be difficult to determine an appropriate testing metric and an efficient technique for obtaining that metric. One metric used to measure the completeness of software testing is the number of lines of code of the software being tested which are executed for a given set of tests. This coverage analysis information enables determination of what lines of code have yet to be exercised to insure sufficiently complete testing. Full statement coverage is when all lines of code have been executed for a given set of tests.
Another related metric for measuring testing completeness is to measure branch coverage for source statements with multiple code paths depending on runtime evaluation of a condition. Specifically, branch coverage is measuring, for a given set of tests, the different code paths exercised in relation to all possible branch alternatives or code paths.
When debugging software, it is typically desirable to have a customizable testing environment allowing a developer to perform data validation for specific routines and parameters being tested, and providing a means for supplying data to drive routines.
It is desirable to provide a system and apparatus for testing software that overcomes the foregoing and other disadvantages of software testing techniques and problems, and which further provide a more efficient means of testing software, insure higher quality software due to more complete and sufficient testing, and save software development time and computer resources. It is to these ends the present invention is directed.